Asymptotic Variance of Passage Time Estimators in Markov Chains

We consider the problem of estimating passage times in stochastic simulations of Markov chains+ Two types of estimator are considered for this purpose: the “simple” and the “overlapping” estimator; they are compared in terms of their asymptotic variance+ The analysis is based on the regenerative structure of the process and it is shown that when estimating the mean passage time, the simple estimator is always asymptotically superior+ However, when the object is to estimate the expectation of a nonlinear function of the passage time, such as the probability that the passage time exceeds a given threshold, then it is shown that the overlapping estimator can be superior in some cases+ Related results in the Reinforcement Learning literature are discussed+